Welcome back. Let's take a look at the code for the request with caching module.
It's not very complicated and it's pretty instructive.
Plus, it's good practice to read
slightly longer code chunks and try to make sense of them.
Let's start here with the definition of the get function.
It takes the same parameters as request.get,
a baseurl, and the optional params dictionary.
But it also has a few extra optional parameters like private_keys_to_ignore,
we'll come back to those.
In the previous video,
I described conceptually how a cache works.
Now we've got a datastore where we're going
to store the results of our previous invocations of request.get.
But how are we actually going to implement that in Python?
The answer is we're going to do it as a dictionary,
where each key will represent one invocation of request.get.
The key has to somehow encode what
the baseurl and what the parameters were of that invocation of requests.get,
and then the results will be some text
that was what we got back from running request.get.
So, in our code here,
we're going to compute a cache key.
We're going to make this value K by calling a function,
a little helper function that I'll show you in a minute called make_cache_key.
It's going to take the baseurl and the params and turn it into a string.
We'll take a look at that make_cache_key function in a minute.
But just think of it for now is that it's producing
some string that we'll use as a key in the cache dictionary.
Now the heart of this get function is this if, elif, else.
So, we're going to check whether this cache key is in either of our two caches,
the page specific cache or the permanent cache.
So, if it finds it in the temporary cache,
let's say it's here,
then we're going to take these results and return them.
So, if we found them in temp_cache,
that is the cache key that we're looking for is in the temp_cache,
then we go get the value associated with that cache key,
and that's what we're going to return.
You may recall that we're trying to produce the same kind of
object that request.get produces.
That was a response object that has not only the text but also the URL.
Now, if we actually call request.get,
this would be the URL that we fetched from the remote server.
In this case, it's going to be the URL that we would have
fetched if we had actually gone to the remote server,
but instead, we'll produce that URL that we would
have used if we had to go out to the remote server.
So, that was computed up here using another little helper function.
If it didn't find it in the temporary cache,
it'll do the same thing,
it'll check in the permanent cache.
If it was in the permanent cash,
return the text and the URL.
If it didn't find it in either of the two caches,
then we will actually call request.get,
passing the baseurl in the params.
We will add those,
whatever the result was,
gets added into the cache, and again,
this is another helper function that we'll look at in a minute called add_to_cache.
So, the basic structure is we calculate what
the cache key ought to be and then we check in the two caches is that key there?
Is it in either spot?
If it's in either spot, we return the results from that cache dictionary,
and if it's not there,
then we call request.get and we add
it to the cache so that it will be there the next time.
The only other little tricky part in here and I glossed over is that we
want these caches to live beyond the current run of the active code window.
So to make it be a little more permanent,
at least to live a little longer than the current run,
we want to save it in a file.
I've shown the cache as if it's just a dictionary,
but actually this dictionary we're going to store in a file.
So, we have to read this dictionary from a file,
and I'll show you this _read_from_file as another little helper function.
So, let's take a look at those helper functions now,
now that we understand the basic structure The first one is this, make_cache_key.
So, it's taking the baseurl and the params dictionary,
and it has to make a string out of that request.
A text string that we can use as a key in the cache dictionary.
Now, the URL itself,
we could just make the full URL out of this base URL and the parameters dictionary,
but we're making something slightly different in
our case because there are some keys that we want to exclude.
In particular, they may be private data that's used for authentication,
and we'll see more about this when we get to the Flickr API.
The other thing that we want to do here is to make sure that
we always get the parameters in the same order.
If there is a couple of different parameters remember this dictionary, the params_d.
The keys could come out in any order,
and if one time we call it we get
the keys in one order and another time we get them in a different order,
we might fail in our lookup trying to find something in the cache.
So, we've created a canonical order.
We alphabetize the keys by sorting them,
and the default order is to sort them in alphabetical order.
Then we go through each of the keys and we just create
a little string that puts together the key and the value.
So, we're getting all of the keys and values from params_d,
and we're sticking them together into
one giant string separated by underscore characters.
So, what we're going to get is something that looks a lot like the full URL,
but might be a little bit different.
Let's look at the other helper functions.
First, let's look at this read_from_file.
Basically, we're just taking
the contents of a file and reading them into a Python dictionary.
So, most of the action is right here,
we open the file and then we read it,
and we call json.loads(res) which takes the text
of the file and turns it into a Python object.
In our case, it's going to be a dictionary.
I've wrapped this in something called try and except.
If you haven't seen that before,
don't worry about it too much,
it's covered in the next course in the specialization.
It just lets the code fail gracefully.
So, if it can't find this file or if it can't load it as JSON,
then we'll just return an empty dictionary.
So it'll fail gracefully.
So, that's reading it from the file and turning it into a Python dictionary.
The other helper function is this add_to_cache.
So, we've got some cash filename and we've got some new
key and results that we want to save into the cache.
So, we first read that dictionary using read_from_file,
we add a key to it,
with the value that has been provided,
and then we just write that back out to the file.
And write_to_file, you can see up here is another helper function
that just opens the file and writes the dictionary out.
So, that's a simple caching pattern that we've implemented.
It's a wrapper around the expensive and unreliable operation request.get.
You can just call request with caching.get,
it returns the same thing as request.get.
If it has a saved previous version of the request it returns those contents,
and if not, then it calls request.get.
It's useful because it helps you avoid problems with rate limits when you're debugging
your code that's working with external rest APIs. I'll see you next time